Bottom-up Deep Learning using the Hebbian Principle

The “fire together, wire together” Hebbian learning model is a central principle in neuroscience, but, surprisingly, it has found limited applicability in modern machine learning. In this paper, we show that neuro-plausible variants of competitive Hebbian learning provide a promising foundation for bottom-up deep learning. We propose an unsupervised learning algorithm termed Adaptive Hebbian Learning (AHL) which trains rapidly, with minimal tuning, while providing sparse, distributed neural codes. We obtain excellent classification results for standard image datasets with deep convolutional nets using AHL as a building block, with a final SVM layer. We also propose a Discriminative Hebbian Learning (DHL) algorithm that can utilize class labels, by combining Hebbian and anti-Hebbian mechanisms in a manner inspired by dopamine modulated learning. We show that combining AHL for lower layers with DHL for higher layers is a promising strategy for deep learning.